Skip to content

fix(bin): detect and repair crewmate panes that survive a restart unable to work - #1421

Open
Moshik21 wants to merge 4 commits into
kunchenguid:mainfrom
Moshik21:fm/fm-restart-autonomy-recovery-r2
Open

fix(bin): detect and repair crewmate panes that survive a restart unable to work#1421
Moshik21 wants to merge 4 commits into
kunchenguid:mainfrom
Moshik21:fm/fm-restart-autonomy-recovery-r2

Conversation

@Moshik21

Copy link
Copy Markdown

Intent

Firstmate crewmate panes that survive a host crash or reboot come back looking perfectly healthy while being silently unable to work, and the existing endpoint-liveness check reports them ALIVE. This happened twice on 2026-07-31 and stopped the fleet both times. The captain asked to make firstmate detect and repair it, and required diagnosing the real mechanism first rather than building on their reconstruction.

Verified mechanism (against herdr 0.7.5 and Claude Code 2.1.220): herdr persists only each pane's CREATION cwd, and its default [session] resume_agents_on_restore relaunches the agent from the integration-reported session ref (claude --resume ). That command is herdr's own, so it carries none of the launch shape fm-spawn resolved. The pane therefore returns without its autonomy flag - stalling forever on its first tool call waiting for an approval nobody is watching for - and in the project checkout, because the task worktree was only ever entered by a treehouse get subshell whose cwd the persisted creation cwd never followed.

Detection: new bin/fm-crew-fitness.sh reports whether a recorded task's pane still carries the autonomy flags its own recorded launch command specified and is still in its recorded worktree. The session-start digest runs it for every live endpoint, because the crash that causes this takes the watcher down too, so session start is where it actually surfaces. Deliberate decisions: the flag set is intersected with the task's own recorded launch rather than mapped per harness, so a template change can never leave the check asserting a stale flag; exact argv-element matching is required rather than a substring scan, because a brief travels in argv and can quote the very flag being looked for; and both scripts fail closed toward 'unknown' rather than 'fit', since a false 'fit' is the exact silent failure being fixed. The endpoint is proven present through the recovery-grade state contract first, because tmux silently resolves an absent target to the caller's own window, which would otherwise condemn a vanished task by measuring firstmate's own pane.

Repair: fm-spawn.sh now records the fully resolved launch command as launch= in the task's metadata so recovery is deterministic and does not re-derive flags under pressure, and new bin/fm-crew-relaunch.sh replays it in the recorded worktree. It refuses to relaunch over a live agent, refuses when the recorded worktree is unaccounted for, never allocates a second worktree, and never touches worktree contents - the three safety properties the captain required be preserved.

The change also corrects a previously recorded learning about why a keystroke cannot repair this: bypassPermissions is a launch-time grant, so a process started without it does not offer that mode at all, independent of any key-transmission question about shift+tab.

Scope was deliberately held to a recovery correctness fix rather than a new supervision subsystem, per the captain's instruction. Firstmate additionally asked me to consider folding in a spawn-time task-vs-task worktree ownership assertion if it were small. I implemented it, then testing against real treehouse and herdr disproved its premise, so I deliberately did NOT ship it and am reporting it back instead - it needs a liveness contract that is larger than this branch should carry.

Covered by tests/fm-crew-fitness.test.sh (44 assertions) proving detection of both the manual-mode case and the wrong-cwd case, and proving the repair restores an autonomous agent in the correct worktree without duplicating a task or losing uncommitted work. This is firstmate shared tracked material, so it follows firstmate-coding-guidelines: one sentence per line in docs, shellcheck-clean scripts via bin/fm-lint.sh, colocated tests, no agent co-author.

What Changed

  • Adds bin/fm-crew-fitness.sh, which reports whether a live task pane still carries the autonomy grants its own recorded launch= specified and is still in its recorded worktree, on a three-state fit|unfit|unknown verdict that fails closed toward unknown; bin/fm-session-start.sh now emits one fitness line per endpoint that read as alive, since the crash that causes this failure takes the watcher down too.
  • Adds bin/fm-crew-relaunch.sh, which replays a task's recorded launch command in its recorded worktree — refusing over a live or vanished endpoint, never allocating a second worktree, never touching worktree contents, and appending a recovery note to the brief first. To make that deterministic, bin/fm-spawn.sh records the fully resolved launch command as launch= in state/<id>.meta and rejects newline-bearing launch strings that would corrupt the metadata.
  • Extends the backend layer with fm_backend_pane_argv_has, fm_backend_pane_env_has, fm_backend_current_path_passive, and dispatchers for send_text_line/current_path, implemented passively for tmux (/proc argv and environ) and herdr (pane process-info) and returning undeterminable elsewhere; covered by tests/fm-crew-fitness.test.sh, tests/fm-backend-herdr.test.sh, and a new tests/fm-spawn-dispatch-profile.test.sh case pinning the recorded launch= line, with the backend limits recorded in docs/verification/runtime-backends.md.

Risk Assessment

✅ Low: Every finding from both prior rounds is fixed with behavior-level test coverage and honest verification evidence, the new env reader preserves the never-toward-fit invariant by construction, and the two trickiest new mechanisms (the array-expansion guard and the shell quote round-trip) verify correct under direct exercise.

Testing

I ran the change's own suite (tests/fm-crew-fitness.test.sh, 66 assertions) plus the adjacent herdr/backend/spawn suites, then demonstrated the intent the way the captain would actually meet it: a real tmux pane running a stand-in harness in a real task worktree with uncommitted work, restored herdr-style as claude --resume &lt;id&gt; in the project checkout. The session-start digest reports that pane endpoint: alive and immediately below it fitness: unfit (autonomy=lost cwd=wrong) naming the lost --dangerously-skip-permissions and the wrong directory; relaunch refuses while the agent is live, dry-run changes nothing, and after exiting the agent the repair brings back an autonomous agent in the recorded worktree (fitness: fit) with the uncommitted file intact, the same worktree count, GOTMPDIR recreated, and a recovery note appended to the brief. I also found and closed a coverage gap — nothing asserted that fm-spawn.sh writes launch= into task metadata, the record both recovery scripts depend on — by adding a focused test that pins the recorded line to the literal command typed into the pane; it fails at the base commit and passes here. This is a terminal CLI surface with no GUI, so the reviewer-visible evidence is the captured tmux pane render and the session-start digest transcript rather than a screenshot. One adjacent suite, tests/fm-session-start.test.sh, fails on this host for a pre-existing fixture reason (real /usr/bin/node in the fixture PATH) and fails identically at the base commit.

Evidence: Restart-recovery end-to-end transcript (session-start digest before/after the crash and after the repair, plus the rendered tmux pane)

================ SCENE 1: the crewmate as it should be ================
session start digest:
--- fix-parser ---
window=fmdemo:fm-fix-parser
worktree=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-parser
project=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project
harness=claude
model=opus
effort=xhigh
kind=ship
mode=no-mistakes
tasktmp=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/tmp-fix-parser
launch=CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'while :; do sleep 1; done' --dangerously-skip-permissions --model opus --effort xhigh 'brief: fix the parser crash'
endpoint: alive (backend=tmux window=fmdemo:fm-fix-parser)
fitness: fit (autonomy=ok cwd=ok)
status tail: (no status file yet: /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/home/state/fix-parser.status)


================ SCENE 2: the host reboots; herdr restores the pane with `claude --resume` ================
what the pane looks like to a human (tmux pane, restored):
  | cd '/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project' && /
  | tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'wh
  | ile :; do sleep 1; done' --resume 21637ebd-8ca7-41bc-9c3b-1b8079c08ed0
  | konne@Scholar:~/.no-mistakes/worktrees/796f20c5af81/01KYWSQ51EXYW8YPM3TQA704SG$
  | cd '/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project' && /
  | tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'wh
  | ile :; do sleep 1; done' --resume 21637ebd-8ca7-41bc-9c3b-1b8079c08ed0

session start digest:
--- fix-parser ---
window=fmdemo:fm-fix-parser
worktree=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-parser
project=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project
harness=claude
model=opus
effort=xhigh
kind=ship
mode=no-mistakes
tasktmp=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/tmp-fix-parser
launch=CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'while :; do sleep 1; done' --dangerously-skip-permissions --model opus --effort xhigh 'brief: fix the parser crash'
endpoint: alive (backend=tmux window=fmdemo:fm-fix-parser)
fitness: unfit (autonomy=lost cwd=wrong) - autonomy grant no longer in force: --dangerously-skip-permissions; pane is in /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project, not the recorded worktree /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-parser
status tail: (no status file yet: /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/home/state/fix-parser.status)


the standalone check, and its exit status:
fitness: unfit (autonomy=lost cwd=wrong) - autonomy grant no longer in force: --dangerously-skip-permissions; pane is in /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project, not the recorded worktree /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-parser
exit=1

fleet-wide sweep:
fix-parser: fitness: unfit (autonomy=lost cwd=wrong) - autonomy grant no longer in force: --dangerously-skip-permissions; pane is in /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project, not the recorded worktree /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-parser
exit=1

================ SCENE 3: the repair ================
relaunching over the live agent is refused:
  error: an agent is still running in task 'fix-parser''s pane; exit it with its adapter's exit command first (harness-adapters skill), then re-run - relaunching over a live agent would put two agents on one task

the reboot also cleared the task tmp root:
  /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/tmp-fix-parser/gotmp exists: no

exiting the stalled agent, as the recovery playbook requires:

dry run:
  would append recovery note to: /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/home/data/fix-parser/brief.md
  would cd pane fmdemo:fm-fix-parser (backend=tmux) to: /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-parser
  would create: /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/tmp-fix-parser/gotmp
  would send env: export GOTMPDIR=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/tmp-fix-parser/gotmp
  would send launch: CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'while :; do sleep 1; done' --dangerously-skip-permissions --model opus --effort xhigh 'brief: fix the parser crash'
  (dry run created /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/tmp-fix-parser/gotmp: no)

repairing:
  relaunched fix-parser harness=claude backend=tmux worktree=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-parser

the repaired pane:
  | cd '/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project' && /
  | tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'wh
  | ile :; do sleep 1; done' --resume 21637ebd-8ca7-41bc-9c3b-1b8079c08ed0
  | konne@Scholar:~/.no-mistakes/worktrees/796f20c5af81/01KYWSQ51EXYW8YPM3TQA704SG$
  | cd '/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project' && /
  | tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'wh
  | ile :; do sleep 1; done' --resume 21637ebd-8ca7-41bc-9c3b-1b8079c08ed0
  | ^C
  | konne@Scholar:/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/pro
  | ject$ cd /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-p
  | arser
  | konne@Scholar:/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-
  | fix-parser$ export GOTMPDIR=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG
  | /demo-work/tmp-fix-parser/gotmp
  | konne@Scholar:/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-
  | fix-parser$ CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false /tmp/no-mistakes-evidence
  | /01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'while :; do sleep 1; done'
  |  --dangerously-skip-permissions --model opus --effort xhigh 'brief: fix the pars
  | er crash'

session start digest:
--- fix-parser ---
window=fmdemo:fm-fix-parser
worktree=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/wt-fix-parser
project=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/project
harness=claude
model=opus
effort=xhigh
kind=ship
mode=no-mistakes
tasktmp=/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/tmp-fix-parser
launch=CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/shim/claude -c 'while :; do sleep 1; done' --dangerously-skip-permissions --model opus --effort xhigh 'brief: fix the parser crash'
endpoint: alive (backend=tmux window=fmdemo:fm-fix-parser)
fitness: fit (autonomy=ok cwd=ok)
status tail: (no status file yet: /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/demo-work/home/state/fix-parser.status)


what the repair did and did not touch:
  uncommitted work still there: half-finished parser work the crewmate must not lose
  worktrees for this project: 2 before, 2 after
  task GOTMPDIR recreated: yes
  brief now ends with:
    
    ## Recovery note
    
    This session was interrupted and relaunched in place. Your worktree, its commits, and its uncommitted changes are exactly as you left them. Inspect the worktree first and continue from that state - do not restart the task from the beginning.
Evidence: The digest lines a captain sees, before the crash / after the herdr restore / after the repair
fitness: fit (autonomy=ok cwd=ok)

endpoint: alive (backend=tmux window=fmdemo:fm-fix-parser)
fitness: unfit (autonomy=lost cwd=wrong) - autonomy grant no longer in force: --dangerously-skip-permissions; pane is in <demo>/project, not the recorded worktree <demo>/wt-fix-parser

relaunched fix-parser harness=claude backend=tmux worktree=<demo>/wt-fix-parser
endpoint: alive (backend=tmux window=fmdemo:fm-fix-parser)
fitness: fit (autonomy=ok cwd=ok)

uncommitted work still there: half-finished parser work the crewmate must not lose
worktrees for this project: 2 before, 2 after
task GOTMPDIR recreated: yes
Evidence: Reproducer for the end-to-end demo (bash demo-restart-recovery.sh <repo-root> <out-dir>)
#!/usr/bin/env bash
# End-to-end demonstration of the restart-recovery fix, from the captain's seat:
#
#   Scene 1 - a healthy crewmate pane, seen at session start.
#   Scene 2 - a herdr-style restart brings the pane back `claude --resume <id>`
#             in the project checkout: endpoint liveness still says ALIVE.
#   Scene 3 - the repair: exit the agent, replay the recorded launch command.
#
# Real tmux (private socket), a real git project + task worktree, real
# uncommitted work in it. The "agent" is a copy of /bin/bash named `claude`, so
# tmux's agent-state read sees a live harness while /proc carries the argv of
# the case being shown. No real harness is launched.
set -u

ROOT=${1:?repo root}
OUT=${2:?evidence dir}

REAL_TMUX=$(command -v tmux)
SOCKET="fm-demo-$$"
WORK="$OUT/demo-work"
rm -rf "$WORK"; mkdir -p "$WORK"

cleanup() { "$REAL_TMUX" -L "$SOCKET" kill-server >/dev/null 2>&1 || true; }
trap cleanup EXIT

mkdir -p "$WORK/shim"
cat > "$WORK/shim/tmux" <<SH
#!/usr/bin/env bash
exec "$REAL_TMUX" -L "$SOCKET" "\$@"
SH
chmod +x "$WORK/shim/tmux"
cp /bin/bash "$WORK/shim/claude"
PATH="$WORK/shim:$PATH"; export PATH

HOME_DIR="$WORK/home"
mkdir -p "$HOME_DIR/state" "$HOME_DIR/data" "$HOME_DIR/config" "$HOME_DIR/projects"
export FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$HOME_DIR/state" \
       FM_DATA_OVERRIDE="$HOME_DIR/data" FM_CONFIG_OVERRIDE="$HOME_DIR/config" \
       FM_PROJECTS_OVERRIDE="$HOME_DIR/projects"

PROJECT="$WORK/project"
mkdir -p "$PROJECT"
git -C "$PROJECT" init -q -b main
git -C "$PROJECT" config user.email demo@example.com
git -C "$PROJECT" config user.name Demo
echo base > "$PROJECT/file.txt"
git -C "$PROJECT" add file.txt
git -C "$PROJECT" commit -qm base
WORKTREE="$WORK/wt-fix-parser"
git -C "$PROJECT" worktree add -q -b fix-parser "$WORKTREE"
printf 'half-finished parser work the crewmate must not lose\n' > "$WORKTREE/wip.txt"

TASKTMP="$WORK/tmp-fix-parser"
mkdir -p "$TASKTMP/gotmp"
mkdir -p "$HOME_DIR/data/fix-parser"
printf '# Brief\n\nFix the parser crash on empty input.\n' > "$HOME_DIR/data/fix-parser/brief.md"

SESSION=fmdemo
"$REAL_TMUX" -L "$SOCKET" new-session -d -s "$SESSION" -n captain

# The launch command fm-spawn.sh resolves and now records as launch=. Here the
# harness binary is the stand-in, but the SHAPE is fm-spawn's: env prefix,
# autonomy flag, model/effort, brief.
LAUNCH="CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false $WORK/shim/claude -c 'while :; do sleep 1; done' --dangerously-skip-permissions --model opus --effort xhigh 'brief: fix the parser crash'"

cat > "$HOME_DIR/state/fix-parser.meta" <<EOF
window=$SESSION:fm-fix-parser
worktree=$WORKTREE
project=$PROJECT
harness=claude
model=opus
effort=xhigh
kind=ship
mode=no-mistakes
tasktmp=$TASKTMP
launch=$LAUNCH
EOF

wait_for() {  # <target> <expected pane_current_command>
  local i=0
  while [ "$i" -lt 100 ]; do
    [ "$("$REAL_TMUX" -L "$SOCKET" display-message -p -t "$1" '#{pane_current_command}' 2>/dev/null)" = "$2" ] && return 0
    sleep 0.1; i=$((i + 1))
  done
  return 1
}

digest() {  # print just the crewmate's slice of the session-start digest
  "$ROOT/bin/fm-session-start.sh" 2>/dev/null \
    | sed -n '/--- fix-parser ---/,/^$/p'
}

banner() { printf '\n================ %s ================\n' "$1"; }

# ---------------------------------------------------------------- scene 1 ----
banner 'SCENE 1: the crewmate as it should be'
"$REAL_TMUX" -L "$SOCKET" new-window -d -t "$SESSION:" -n fm-fix-parser
"$REAL_TMUX" -L "$SOCKET" send-keys -t "$SESSION:fm-fix-parser" "cd '$WORKTREE' && $LAUNCH" Enter
wait_for "$SESSION:fm-fix-parser" claude || echo 'demo error: agent did not start'
printf 'session start digest:\n'
digest

# ---------------------------------------------------------------- scene 2 ----
banner 'SCENE 2: the host reboots; herdr restores the pane with `claude --resume`'
"$REAL_TMUX" -L "$SOCKET" kill-window -t "$SESSION:fm-fix-parser"
"$REAL_TMUX" -L "$SOCKET" new-window -d -t "$SESSION:" -n fm-fix-parser
# Exactly what herdr's resume_agents_on_restore produces: its own resume
# command, from the persisted CREATION cwd (the project checkout), carrying
# none of firstmate's launch shape.
"$REAL_TMUX" -L "$SOCKET" send-keys -t "$SESSION:fm-fix-parser" \
  "cd '$PROJECT' && $WORK/shim/claude -c 'while :; do sleep 1; done' --resume 21637ebd-8ca7-41bc-9c3b-1b8079c08ed0" Enter
wait_for "$SESSION:fm-fix-parser" claude || echo 'demo error: restored agent did not start'
printf 'what the pane looks like to a human (tmux pane, restored):\n'
"$REAL_TMUX" -L "$SOCKET" capture-pane -p -t "$SESSION:fm-fix-parser" | sed '/^$/d' | sed 's/^/  | /'
printf '\nsession start digest:\n'
digest
printf '\nthe standalone check, and its exit status:\n'
"$ROOT/bin/fm-crew-fitness.sh" fix-parser; printf 'exit=%s\n' "$?"
printf '\nfleet-wide sweep:\n'
"$ROOT/bin/fm-crew-fitness.sh" --all; printf 'exit=%s\n' "$?"

# ---------------------------------------------------------------- scene 3 ----
banner 'SCENE 3: the repair'
printf 'relaunching over the live agent is refused:\n'
"$ROOT/bin/fm-crew-relaunch.sh" fix-parser 2>&1 | sed 's/^/  /'

printf '\nthe reboot also cleared the task tmp root:\n'
rm -rf "$TASKTMP"
printf '  %s exists: %s\n' "$TASKTMP/gotmp" "$([ -d "$TASKTMP/gotmp" ] && echo yes || echo no)"

printf '\nexiting the stalled agent, as the recovery playbook requires:\n'
"$REAL_TMUX" -L "$SOCKET" send-keys -t "$SESSION:fm-fix-parser" C-c
i=0
while [ "$i" -lt 100 ]; do
  case "$("$REAL_TMUX" -L "$SOCKET" display-message -p -t "$SESSION:fm-fix-parser" '#{pane_current_command}' 2>/dev/null)" in
    claude) sleep 0.1; i=$((i + 1)) ;;
    *) break ;;
  esac
done

printf '\ndry run:\n'
"$ROOT/bin/fm-crew-relaunch.sh" fix-parser --dry-run 2>&1 | sed 's/^/  /'
printf '  (dry run created %s: %s)\n' "$TASKTMP/gotmp" "$([ -d "$TASKTMP/gotmp" ] && echo yes || echo no)"

printf '\nrepairing:\n'
WT_BEFORE=$(git -C "$PROJECT" worktree list | wc -l)
"$ROOT/bin/fm-crew-relaunch.sh" fix-parser 2>&1 | sed 's/^/  /'
wait_for "$SESSION:fm-fix-parser" claude || echo 'demo error: repaired agent did not start'

printf '\nthe repaired pane:\n'
"$REAL_TMUX" -L "$SOCKET" capture-pane -p -t "$SESSION:fm-fix-parser" | sed '/^$/d' | sed 's/^/  | /'
printf '\nsession start digest:\n'
digest

printf '\nwhat the repair did and did not touch:\n'
printf '  uncommitted work still there: %s\n' "$(cat "$WORKTREE/wip.txt" 2>/dev/null)"
printf '  worktrees for this project: %s before, %s after\n' "$WT_BEFORE" "$(git -C "$PROJECT" worktree list | wc -l)"
printf '  task GOTMPDIR recreated: %s\n' "$([ -d "$TASKTMP/gotmp" ] && echo yes || echo no)"
printf '  brief now ends with:\n'
tail -4 "$HOME_DIR/data/fix-parser/brief.md" | sed 's/^/    /'
Evidence: fm-crew-fitness suite output (detection + repair assertions)
FM_TEST_BEGIN 2026-07-31T20:19:10Z tests/fm-crew-fitness.test.sh family=backend-dispatch expected_gate_skip=none
ok - healthy pane reports fit
ok - healthy pane exits 0
ok - manual-mode pane reports unfit
ok - manual-mode pane names the autonomy axis
ok - manual-mode detail names the lost flag
ok - manual-mode pane exits 1
ok - wrong-cwd pane reports unfit
ok - wrong-cwd pane names the cwd axis
ok - wrong-cwd detail names where the pane actually is
ok - wrong-cwd pane exits 1
ok - combined case reports lost autonomy
ok - combined case reports wrong cwd
ok - a brief quoting the flag is not mistaken for the flag itself
ok - a launch with no recognized autonomy flag reports n/a
ok - n/a autonomy with a correct cwd is still fit
ok - n/a autonomy with a correct cwd exits 0
ok - a dropped env-prefix grant is detected as lost, not excused as n/a
ok - the detail names the env-prefix grant that is gone
ok - a dropped env-prefix grant exits 1
ok - an env-prefix grant still in force reads ok, not a permanent unknown
ok - an env-prefix grant still in force exits 0
ok - a secondmate that lost its FM_HOME prefix is unfit even with its flag intact
ok - the detail names the lost secondmate identity prefix
ok - a secondmate that lost its identity prefix exits 1
ok - an undecodable recorded grant is unknown
ok - an undecodable recorded grant is never claimed to be lost
ok - an undecodable recorded grant never reports fit
ok - an undecodable recorded grant exits 3
ok - a backend that cannot read an environment answers undeterminable, never absent
ok - a malformed environment entry is undeterminable, never a verdict
ok - a task with no recorded launch command reports unknown
ok - a task with no recorded launch command never reports fit
ok - unknown exits 3
ok - an unrecorded task reports unknown
ok - an unrecorded task exits 3
ok - a task with no recorded endpoint reports unknown
ok - a vanished endpoint reports unknown, not a lost flag
ok - a vanished endpoint does not claim the flag was lost
ok - --all labels each task
ok - --all reports healthy tasks too
ok - --all exits 1 when any task is unfit
ok - relaunch refuses while an agent still owns the task
ok - relaunch refusal over a live agent exits 1
ok - relaunch refuses when the recorded worktree is gone
ok - relaunch refusal for a missing worktree exits 1
ok - relaunch created no replacement worktree
ok - relaunch refuses without a recorded launch command
ok - relaunch refusal without a launch command exits 1
ok - relaunch refuses a vanished endpoint as a respawn
ok - relaunch refusal for a vanished endpoint exits 1
ok - a refused relaunch leaves the brief exactly as it was
ok - the pane to be repaired starts unfit
ok - --dry-run does not touch the brief
ok - relaunch succeeds once no agent owns the task
ok - relaunch reports the task it restored
ok - the repaired pane is autonomous and in its worktree
ok - uncommitted work in the worktree survives the repair
ok - uncommitted edits survive the repair
ok - the repair allocates no second worktree for the task
ok - the repair leaves a recovery note in the brief
ok - the repair preserves the original brief
ok - --dry-run on a task with a recorded tasktmp exits 0
ok - --dry-run names the GOTMPDIR export it would replay
ok - --dry-run creates nothing
ok - the repair recreates the task GOTMPDIR the reboot removed
ok - relaunch into a worktree path containing a space succeeds
ok - the relaunched command runs in the recorded worktree even when its path contains a space
ok - the relaunched agent inherits the task GOTMPDIR fm-spawn.sh sets at launch
ok - uncommitted work in a spaced worktree survives the repair
all fm-crew-fitness tests passed
FM_TEST_END 2026-07-31T20:19:15Z tests/fm-crew-fitness.test.sh exit=0 duration_ms=4164 gate_skip=false
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=0 duration_ms=4193
FM_TEST_SUMMARY_FAMILY family=backend-dispatch count=1 duration_ms=4164 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-crew-fitness.test.sh duration_ms=4164
Evidence: Spawn suite output including the added launch= recording test
FM_TEST_BEGIN 2026-07-31T20:24:58Z tests/fm-spawn-dispatch-profile.test.sh family=backend-dispatch expected_gate_skip=none
ok - no --model/--effort records defaults and types the claude launch instructions
ok - spawn records the exact resolved launch command as a single launch= line
ok - relative home overrides ignore CDPATH and become absolute before spawn launch construction
ok - FM_HOME defaults resolve relative paths and preserve absolute spellings
ok - absolute override spellings are preserved in spawn launch paths
ok - unresolvable relative spawn overrides fail with named diagnostics
ok - active crew-dispatch profile requires an explicit harness for ship spawns
ok - active crew-dispatch profile requires an explicit harness for scout spawns
ok - active crew-dispatch profile allows an explicit resolved harness
ok - active crew-dispatch profile allows the legacy positional harness form
ok - active crew-dispatch profile allows the raw launch-command escape hatch
ok - claude receives --model and --effort profile flags
ok - codex receives --model and model_reasoning_effort profile flags
ok - codex omits unsupported max effort instead of passing a bad config value
ok - grok receives --model and --reasoning-effort profile flags
ok - grok omits unsupported max reasoning effort
ok - grok omits unsupported xhigh reasoning effort
ok - opencode receives --model and omits the unsupported effort axis
ok - pi receives --model and --thinking max profile flags
ok - pi-signed shares Pi launch semantics while preserving its configured and recorded identity
ok - pi-signed refuses safely and actionably when the selected executable is unavailable
ok - pi-signed is a distinct persistent secondmate runtime with shared Pi supervision semantics
ok - batch dispatch forwards shared --harness, --model, and --effort to every pair
ok - claude forwards firstmate's CLAUDE_CONFIG_DIR so the crewmate uses the same credential store
ok - claude omits the config-dir prefix when firstmate runs with the single-store default
ok - non-claude harnesses do not receive the claude CLAUDE_CONFIG_DIR prefix
ok - active crew-dispatch profile does not block secondmate launches
# all fm-spawn-dispatch-profile tests passed
FM_TEST_END 2026-07-31T20:25:41Z tests/fm-spawn-dispatch-profile.test.sh exit=0 duration_ms=42636 gate_skip=false
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=0 duration_ms=42666
FM_TEST_SUMMARY_FAMILY family=backend-dispatch count=1 duration_ms=42636 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-spawn-dispatch-profile.test.sh duration_ms=42636
- Evidence: Adjacent backend/spawn/session-start suite output (local file: /tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/adjacent-suites.log) - Outcome: ⚠️ 1 info across 1 run (8m23s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ⚠️ bin/fm-crew-relaunch.sh:139 - bin/fm-backend.sh:730 documents fm_backend_send_text_line's 4th argument as [session], but bin/backends/zellij.sh:470 and bin/backends/cmux.sh:509 take it as [expected-label] (the task label fm-<id>), which their target_ready uses to verify the tab/workspace title. fm-spawn.sh passes $W (the label) so it stays correct, but bin/fm-crew-relaunch.sh:139 and :142 pass $WINDOW, the recorded window= target string (e.g. "<zellij-session>:<pane-id>"). fm_backend_zellij_target_ready then compares that against the tab label, fails, and the send returns 1 — so on zellij and cmux the relaunch always dies at line 140 with "could not send the directory change", after having already appended the recovery note to the brief. Fix: pass "fm-$ID" as the label from both new scripts and rename the dispatcher parameter to [expected-label].
  • ⚠️ bin/fm-crew-fitness.sh:210 - bin/fm-crew-fitness.sh:210 calls fm_backend_current_path with $window as the third argument, which zellij (bin/backends/zellij.sh:403) and cmux (bin/backends/cmux.sh:448) treat as expected-label, so target_ready fails and the cwd axis silently reads unknown on every zellij/cmux task. Worse, those two readers are not passive: they send printf marker; pwd; printf marker into the pane and capture it, and their headers state they are deliberately scoped to fm-spawn's pre-launch worktree poll where injecting a command is acceptable. Correcting the label alone would therefore make fitness submit a bogus prompt into every live crewmate's composer at every session start, contradicting the script's own "Read-only and side-effect free" contract at bin/fm-crew-fitness.sh:218. Fix at the shared boundary: read the cwd axis only from backends with a passive foreground-cwd reader (tmux, herdr) and report cwd=unknown for the rest.
  • ⚠️ bin/fm-crew-relaunch.sh:104 - bin/fm-crew-relaunch.sh:104 licenses a relaunch when fm_backend_agent_state reports missing, but missing means the recorded endpoint is authoritatively absent — there is no pane to send to. I verified tmux send-keys -t &lt;session&gt;:&lt;absent-window&gt; exits 1 with "can't find window" (no misdelivery), and herdr's send_text_line fails its target_ready, so the run always dies at line 140 with "could not send the directory change" — after the recovery note has already been appended to the brief at line 130. The header and stuck-crewmate-recovery both advertise missing as a supported repair state, which sends an operator down a path that cannot succeed. Either refuse missing with an explicit "the endpoint is gone; this is a respawn, not a relaunch" message, or recreate the endpoint before sending.
  • ⚠️ bin/fm-crew-relaunch.sh:142 - fm-spawn.sh:1687 unconditionally sends export GOTMPDIR=$TASK_TMP/gotmp into the pane shell before every launch, so the agent and all its children inherit it. bin/fm-crew-relaunch.sh replays only launch=, and after a host restart the pane shell is fresh, so the relaunched agent runs without GOTMPDIR while the script reports a successful relaunch. The meta already records tasktmp=, so the fix is to send the same export line before the launch command, mirroring fm-spawn.
  • ⚠️ bin/fm-crew-relaunch.sh:139 - bin/fm-crew-relaunch.sh:139 sends cd $WORKTREE with the path interpolated unquoted into the shell command line. A recorded worktree path containing a space or a glob character makes the pane run cd on a truncated path; the send itself still succeeds, so the launch command is then typed in the wrong directory and the script prints "relaunched ..." — reproducing exactly the wrong-worktree failure it exists to repair. Shell-quote the path (printf %q) in the sent command.
  • ⚠️ bin/fm-crew-fitness.sh:189 - The intent requires that "both scripts fail closed toward 'unknown' rather than 'fit', since a false 'fit' is the exact silent failure being fixed", and the header at bin/fm-crew-fitness.sh:212 repeats "never toward fit". But at line 189 a recorded launch whose autonomy grant is an env prefix reports autonomy=n/a, which the verdict rules treat as satisfied. That is reachable as a false fit: opencode's grant is OPENCODE_CONFIG_CONTENT={&#34;permission&#34;:{&#34;*&#34;:&#34;allow&#34;}} (fm-spawn.sh:474), and a herdr resume drops env prefixes exactly as it drops flags. For any secondmate the recorded worktree equals the pane's creation cwd (fm-spawn.sh:806, WT="$PROJ_ABS"), so cwd reads ok after a restart — an opencode or pi secondmate therefore reports fitness: fit while it has lost both its permission grant and its FM_HOME/FM_*_OVERRIDE prefixes. Consider reporting unknown rather than n/a when the recorded launch carries a recognized env-prefix grant.
  • ℹ️ bin/fm-spawn.sh:1685 - bin/fm-spawn.sh:1685 appends launch= verbatim. The comment asserts "One line, no newlines: every launch template is a single command line", which holds for the templates but not for the raw-launch-command escape hatch at fm-spawn.sh:503 (LAUNCH=$ARG3, any caller-supplied string containing a space). A newline in that string writes extra lines into state/<id>.meta, and because fm_meta_get (bin/fm-backend.sh:341) takes tail -1, an injected trailing worktree= or window= line wins over the genuine one for every consumer, including teardown. Reject or strip newlines before recording.
  • ℹ️ bin/fm-backend.sh:924 - fm_backend_pane_argv_has declares "1 - confidently absent" as a contract-level verdict, but its tmux reader resolves the pane through tmux display-message -p -t &lt;target&gt; &#39;#{pane_tty}&#39;, which I confirmed silently succeeds (rc=0) against an absent target and answers about the caller's own pane. The membership guard that makes the verdict trustworthy lives only in the one caller (fm-crew-fitness.sh:161's fm_backend_agent_state gate), not at the shared boundary where the contract is stated. No second caller exists today, so nothing is broken now; worth either folding the exact-window check into the primitive or stating the caller obligation in the contract comment before the next caller appears.
  • ℹ️ bin/backends/herdr.sh:1903 - fm_backend_herdr_pane_argv_has depends on a herdr CLI surface (pane process-info and the .result.process_info.foreground_processes[].argv schema) that appears nowhere else in the repo, has no entry in docs/verification/runtime-backends.md alongside the other verified herdr commands, and is not exercised by tests/fm-crew-fitness.test.sh (which skips without tmux and only drives the tmux adapter). herdr is the backend the two production incidents happened on, so this is the path most likely to be exercised first. Per firstmate-coding-guidelines ("update the corresponding verification evidence when behavior changes"), record the command, version, and exact output backing the argv-array claim.

🔧 Fix: harden restart repair sends and fail-closed fitness verdicts
2 issues (1 warning, 1 info) still open:

  • ⚠️ bin/fm-crew-relaunch.sh:140 - bin/fm-crew-relaunch.sh:140 replays export GOTMPDIR=&lt;tasktmp&gt;/gotmp but never creates the directory. fm-spawn.sh:1349-1355 is explicit that "Go won't create GOTMPDIR, so mkdir before it is used", and TASK_TMP is /tmp/fm-&lt;id&gt; — which the host reboot this script exists to recover from typically clears (systemd-tmpfiles or a tmpfs /tmp). Concrete failure: crewmate reboots, operator runs fm-crew-relaunch.sh, the agent comes back with GOTMPDIR pointing at a path that no longer exists, and its first go build/go test fails with a work-dir creation error instead of falling back to the default temp — a new breakage introduced by the replay, in precisely the scenario it targets. Fix: mkdir -p &#34;$TASKTMP/gotmp&#34; before sending (or fold the mkdir into the sent line), mirroring fm-spawn.sh:1355.
  • ℹ️ bin/fm-crew-fitness.sh:96 - bin/fm-crew-fitness.sh:96 justifies the env-grant cap with "NO backend can verify: a running process's environment is not readable through any session-provider surface". That holds for herdr (its process-info schema exposes argv/argv0/cmdline/cwd/name/pid and no environment), but not for the tmux adapter as implemented: bin/backends/tmux.sh:119-131 already reads /proc/<pid>/cmdline for the same foreground pids, and /proc/<pid>/environ is readable for a same-uid process (Yama's ptrace_scope gate applies to ATTACH, not READ). The practical consequence of the blanket rule is that every secondmate (all carry the FM_HOME prefix) and every opencode task now permanently reports fitness: unknown at every session start, healthy or not — fail-closed as instructed, but a whole task class whose verdict can never be actionable. Worth either softening the claim to name the backends it is true for, or adding a passive environ read on tmux under the same three-state contract so tmux secondmates get a real verdict.

🔧 Fix: verify env-prefix grants and recreate replayed GOTMPDIR
1 info still open:

  • ℹ️ bin/backends/tmux.sh:171 - Both tmux readers gate on /proc (bin/backends/tmux.sh:129 for argv, :171 for environ), so on a host with no /proc - which docs/verification/runtime-backends.md:147 now states plainly is every macOS host - the autonomy axis returns undeterminable for every tmux task and fitness can only ever report unknown there. The cwd axis still works via #{pane_current_path}, and herdr (the backend both incidents happened on) answers argv through its own CLI, so the detection this branch exists for does fire where it was needed. Noting the boundary only for the record: it is fail-closed, honestly documented in the verification entry rather than asserted away, and needs no change here.
⚠️ **Test** - 1 info
  • ℹ️ tests/fm-session-start.test.sh:789 - Pre-existing, host-dependent failure unrelated to this change: tests/fm-session-start.test.sh::test_output_ordering_diagnostics_lead fails with "MISSING diagnostic did not appear at all". The case deletes the fake node to force a MISSING diagnostic, but this host has /usr/bin/node, which is inside the fixture's BASE_PATH (/usr/bin:/bin:/usr/sbin:/sbin), so bootstrap still finds node. Verified it fails identically at the base commit 9fdef64, so it is not caused by this branch; no action taken.
  • bash bin/fm-test-run.sh tests/fm-crew-fitness.test.sh — 66 assertions covering the manual-mode case, wrong-cwd case, combined crash shape, quoted-flag argv matching, env-prefix grants, fail-closed unknown paths, and the repair (refusals plus the successful relaunch)
  • bash bin/fm-test-run.sh tests/fm-backend-herdr.test.sh tests/fm-backend.test.sh tests/fm-spawn-dispatch-profile.test.sh tests/fm-session-start.test.sh — new herdr pane_argv_has three-state cases and backend dispatch; session-start failure is pre-existing (see findings)
  • Manual end-to-end demo on a real tmux server (private socket) with a real git project + task worktree holding uncommitted work: bin/fm-session-start.sh digest before the crash, after a herdr-style claude --resume &lt;id&gt; restore from the project checkout, and after repair — captured in restart-recovery-transcript.txt
  • bin/fm-crew-fitness.sh fix-parser and bin/fm-crew-fitness.sh --all against the restored pane (exit=1, autonomy=lost cwd=wrong)
  • bin/fm-crew-relaunch.sh fix-parser over a live agent (refused), --dry-run (creates nothing), and the real repair after exiting the agent — then re-checked fitness, uncommitted work, git worktree list count, GOTMPDIR recreation, and the brief's recovery note
  • New tests/fm-spawn-dispatch-profile.test.sh::test_meta_records_the_exact_launch_command_that_was_typed — pins launch= in state/<id>.meta to the literal command sent to the pane, as a single line, carrying the autonomy grant fitness reads
  • Mutation check: ran the new spawn test against a scratch checkout of base commit 9fdef64 — it fails there ("spawn recorded no launch= line") and passes on the target, confirming it detects the absence of the recording
⚠️ **Document** - 2 infos
  • ℹ️ The intent reports a spawn-time task-vs-task worktree ownership assertion that was implemented, disproved against real treehouse/herdr, and deliberately not shipped because it needs a larger liveness contract. I deliberately did not create a documentation home for it: the placement policy forbids new surfaces and incident narratives for work that is not in the tree, and there is no invariant in this branch to own. If the captain wants it durable, the right follow-up is a backlog item, not a tracked doc.
  • ℹ️ tests/fm-spawn-dispatch-profile.test.sh - This file was already modified and uncommitted in the worktree when this phase started (it is not part of the base..target diff and I did not touch it). Flagging so the outer executor knows the working tree carries a test edit outside the documented change.
✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Moshik21 added 4 commits July 31, 2026 12:16
A session-provider restart can bring a crewmate pane back looking
perfectly healthy while it is silently unable to work, and endpoint
liveness reports it ALIVE. This happened twice on 2026-07-31 and stopped
the fleet both times.

Mechanism, verified against herdr 0.7.5 and Claude Code 2.1.220: herdr
persists only each pane's CREATION cwd, and its default
`[session] resume_agents_on_restore` relaunches the agent from the
integration-reported session ref (`claude --resume <id>`). That command
is herdr's own, so it carries none of the launch shape fm-spawn
resolved. The pane returns without its autonomy flag - stalling forever
on its first tool call - and in the project checkout, because the task
worktree was only ever entered by a `treehouse get` subshell the
persisted creation cwd never followed.

Detection: fm-crew-fitness.sh reports whether a recorded task's pane
still carries the autonomy flags its own recorded launch command
specified and is still in its recorded worktree. The session-start
digest runs it for every live endpoint - the crash that causes this
takes the watcher down too, so a session start is where it surfaces.
The flag set is intersected with the task's recorded launch rather than
mapped per harness, so a template change can never leave the check
asserting a stale flag. Exact argv-element matching is required, not a
substring scan: a brief travels in argv and can quote the very flag
being looked for.

Repair: fm-spawn.sh now records the fully resolved launch command as
`launch=` in the task's metadata, and fm-crew-relaunch.sh replays it in
the recorded worktree. It refuses to relaunch over a live agent, refuses
when the recorded worktree is unaccounted for, never allocates a second
worktree, and never touches worktree contents.

Both fail closed toward "unknown": a false "fit" is the exact silent
failure being fixed. The endpoint is proven present through the
recovery-grade state contract first, because tmux silently resolves an
absent target to the caller's own window - which would otherwise
condemn a vanished task by measuring firstmate's own pane.

Also corrects the recorded reason a keystroke cannot repair this:
bypassPermissions is a launch-time grant, so a process started without
it does not offer that mode at all, independent of key transmission.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant